{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AIRPRICE Prediction- Stavya Bhatia - 0.9569.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "nyCFPUQSsFlP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# AIRPRICE PREDICTION - Approach"
      ]
    },
    {
      "metadata": {
        "id": "OL1o_E8QsFlR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The approach taken in the modelling of airplane price prediction is as follows:\n",
        "\n",
        "1) Data exploration: \n",
        "    I familiarized myself with the train and test data set. This included understanding the number of rows and columns of train and test data, the type of data in each column, the identification of missing values if any, finding out the value distribution of columns, and finally understanding variation of price with individual columns using visualization.\n",
        "   \n",
        "2) Data treatment and Feature Generation:\n",
        "    After a thorough EDA that helped me familiarize myself with the train and test data set, I decided to coalesce the two data sets to fix some common data issues. For eg: I noted that arrival_time column had an incorrect date value whenever the flight travelled from one date into another- I decided to fix this by adding duration to the departure date time to get the correct arrival date time. I will talk about these more in detail at the time of coding for better understanding.\n",
        "    \n",
        "3) Target variable Normalization:\n",
        "    As noted in most regression models, the models fit better when the target variable is normalized. I used the seaborn package to develop a distribution plot and scipy to develop a probability plot to understand the price distribution in the data. I found the data to have a little right skew with ireegularities and found that the logarithmic function helps me normalize data to a certain extent. \n",
        "\n",
        "    Please note, I do not believe this was the best treatment for normalizing pricing, however, I chose to go with it after trying some other normalization methods that didnt give a better output than log, but I am certain there might be a better value treatment I currently havent discovered.\n",
        "    \n",
        "4) Null Value Treatment:\n",
        "    I identified only two missing values overall which were in the columns route and total_stops. On exloring a little further I found both the values to be missing in the same row. I did some digging into the data by filtering other columns that had the same value as the row with missing values. For instance, I filtered the data for all rows having the following column values:\n",
        "    Destination - Cochin\n",
        "    Source - Delhi\n",
        "    Duration - 1420 min\n",
        "    Month - 5\n",
        "    Airline - AirIndia\n",
        " The idea behind this filtering was that a flight flying between the same source and destination, having the same duration and carrier in the same month will have a similiar route. (This was a hypothesis vaguely based on a paper I read on flight scheduling)\n",
        " I found a few other flights with the same column values that had the same route and total stops validating my hypothesis. I replicated the same values for my missing row.\n",
        " \n",
        "5) Outlier Treatment:\n",
        "    Post missing value treatment I moved further to remove any outliers in my data. I used two models seperately namely ElasticNet and Ridge to fit my training data's dependent and independent variables. I then calculated residuals of both the fits individually. I then calculated Z value of each point by subtracting individual residual values with the mean residual value and then dividing it by the standard deviation of residual value. Any Z value greater than 3 standard deviation of the Z values was considered an outlier for that model. The outlier rows common to both the models were considered as outliers of my main model. \n",
        "    \n",
        "    Please note, using this technique I saw my score drop rather than increase so I decided to not use it as a part of my code. I have only explained this as it was a part of my thought process for solving the problem. I have left the code in comments below for reference.\n",
        "\n",
        "6) Skew Treatment:\n",
        "    I used boxcox transformation to treat the skewness of my independent variables to help better fit my model. I learnt this following technique from learning material on Kaggle.\n",
        "    \n",
        "7) Modeling:\n",
        "    I built a stack regressor that uses LightGBM regressor, GradientBoosting regressor, SVR , Random Forest regressor, and Kernel ridge as base models and XGboost regressor as my meta model. The process was to first select individual models and hyperparameter tune them using GridSearchCV (with assistance from articles on analytics vidhya), then check the root mean squared value using a function 'rmsle_cv' that I developed. This function uses the KFold technique for cross validation that helps prevent overfitting on the train data.\n",
        "  Once all the models were hyperparameter tuned and the individual RMSE scores were satisfactory I stacked them for my final price output.\n",
        "  \n",
        "Please Note:\n",
        "\n",
        "    The code was run iteratively and a lot of things I did to reach this particular section have been removed during the process for simple understanding. For example, GridSearchCV was used for parameter tuning but was removed once the paramaters were tuned satisfactorily. Similiarly, null value treatment consisted of several group by's and column filtering before I finally reached at the column values mentioned above, but have since been removed. \n",
        "\n",
        "    Most of the data treatment, feature generation, skew treatment was a part of an iterative process that were edited/omitted/added on the bases of how they scored on the leaderboard. So there is no smooth top down approach to coding that I took, however, I have tried to keep it as clean as possible.\n",
        "\n",
        "    Outlier treatment was removed from the model but has been commented in the code for understanding\n",
        "    \n",
        "    Other models I tried- Lasso, Ridge, Elasticnet , Keras Neural Network regressor, Cat Boost, Blending models.\n",
        "    "
      ]
    },
    {
      "metadata": {
        "id": "f6tpaJ5esFlS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Importing packages for data processing, feature generation, outlier detection, missing value treatment, skew treatment and modelling"
      ]
    },
    {
      "metadata": {
        "id": "kiEcjq1SsFlT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd \n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from scipy import stats\n",
        "from sklearn.model_selection import KFold\n",
        "#from sklearn.linear_model import LassoCV , RidgeCV, ElasticNet, Lasso,Ridge\n",
        "from sklearn.cross_validation import cross_val_score\n",
        "from sklearn.pipeline import make_pipeline, Pipeline\n",
        "from sklearn.preprocessing import RobustScaler\n",
        "from sklearn.ensemble import GradientBoostingRegressor\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from lightgbm import LGBMRegressor\n",
        "from xgboost import XGBRegressor  \n",
        "from sklearn.svm import SVR\n",
        "from sklearn.metrics import mean_squared_error\n",
        "import datetime\n",
        "from mlxtend.regressor import StackingCVRegressor\n",
        "from sklearn.kernel_ridge import KernelRidge\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from scipy.stats import skew\n",
        "from scipy.special import boxcox1p\n",
        "from scipy.stats import boxcox_normmax"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RZ6dBbVksFlY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data import and EDA"
      ]
    },
    {
      "metadata": {
        "id": "x7ow4izesFlY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train=pd.read_excel(\"Data_Train.xlsx\")\n",
        "test=pd.read_excel(\"Test_set.xlsx\")\n",
        "sample=pd.read_excel(\"Sample_submission.xlsx\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "Fg7dm25GsFld",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nip96KmHsFlf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train.info()\n",
        "train.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "MwFmqkaesFlh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "test.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1wN2UCubsFlk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "test.info()\n",
        "test.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CpLnLb0usFlo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# storing number of rows as an a variable for later use\n",
        "len_train=train.shape[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "baRrYFznsFlr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#coalescing the two columns for data treatment\n",
        "train=train.append(test)\n",
        "train=train.reset_index()\n",
        "train=train.drop('index',axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9DYuP1resFlu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Visually understanding the variation of price with different columns\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "sns.boxplot(train.Airline, train.Price)\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "sns.boxplot(train.Total_Stops, train.Price)\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "plt.plot(train.sort_values(['Date_of_Journey']).Date_of_Journey.unique(), train.groupby(['Date_of_Journey']).mean())\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "plt.scatter(train.Date_of_Journey, train.Price)\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "sns.boxplot(train.Source, train.Price )\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "sns.boxplot(train.Destination, train.Price )\n",
        "\n",
        "plt.subplots(figsize=[15,5])\n",
        "plt.xticks(rotation=90)\n",
        "sns.barplot(train.Additional_Info, train.Price )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yN11oJ4QsFlx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data Treatment and Feature Generation"
      ]
    },
    {
      "metadata": {
        "id": "zoV2rv5qsFly",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"train['Depart_Time_Hour'] = pd.to_datetime(train.Dep_Time).dt.hour\n",
        "train['Depart_Time_Minutes'] = pd.to_datetime(train.Dep_Time).dt.minute\n",
        "\n",
        "train['Arr_Time_Hour'] = pd.to_datetime(train.Arrival_Time).dt.hour\n",
        "train['Arr_Time_Minutes'] = pd.to_datetime(train.Arrival_Time).dt.minute\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2G1lZ5_ssFl0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting Dep_Time from object (string) to time format\n",
        "train.Dep_Time=pd.to_datetime(train.Dep_Time, format= '%H:%M').dt.time"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pF7xho1ssFl4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# String manipulation to extract only hours and minutes from arrival_time column and dropping the date part, then \n",
        "# converting the string to time format as done above\n",
        "train.Arrival_Time=train.Arrival_Time.str.slice(0,6)\n",
        "train.Arrival_Time=train.Arrival_Time.str.strip()\n",
        "train.Arrival_Time=pd.to_datetime(train.Arrival_Time, format = '%H:%M').dt.time"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-PEB0FNWsFl6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting date of journey to date format\n",
        "train.Date_of_Journey=pd.to_datetime(train.Date_of_Journey,format='%d/%m/%Y').dt.date"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": false,
        "id": "ydI4aI71sFl8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting duration from string to integer value in minutes\n",
        "minute=train['Duration'].str.slice(3,).str.extract('(\\d+)')\n",
        "hour=train['Duration'].str.extract('(\\d+)').astype('int')\n",
        "minute=minute.fillna(0) #filling up minute= 0 where the flight duration only had hours\n",
        "train['Duration']=minute.astype('int')+hour*60"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "guB1VglLsFl_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating column date of arrival by adding date and time of journey with duration in minutes\n",
        "train['Date_of_Arrival']=train.apply(lambda r : pd.datetime.combine(r['Date_of_Journey'],r['Dep_Time']),1) + pd.to_timedelta(train.Duration,unit='m')\n",
        "train['Date_of_Arrival']=train.Date_of_Arrival.dt.date"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": false,
        "id": "leIbLfABsFmB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Changing Total Stops from a string value to a categorical value \n",
        "train.Total_Stops.value_counts()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FTjo42rTsFmE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dic={'non-stop':0,'2 stops':2,'1 stop':1,'3 stops':3,'4 stops': 4}\n",
        "train['Total_Stops']=train.Total_Stops.map(dic)\n",
        "train.Total_Stops=train.Total_Stops.astype('category')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7zV3HetOsFmJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating a column a categorical column class from the column Airline \n",
        "r=train.Airline.str.split(' ',expand=True).get(2)\n",
        "d={None:'Economy','Premium': 'Premium Economy','economy':'Premium Economy','Business':'Business' }\n",
        "r=r.map(d)\n",
        "train['Class']=r\n",
        "train.Class=train.Class.astype('category')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ohy-L0IYsFmN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# The column Additional info had values No info and No Info stored, editing to bring them to the same string value\n",
        "train.loc[train.Additional_Info==\"No info\",'Additional_Info']=\"No Info\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RL_rBUKmsFmU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating an integer column same_day_travel that highlights whether the flight was arrived the same day, a day from dep, or more..\n",
        "train['Same_day_travel']=train.Date_of_Arrival-train.Date_of_Journey\n",
        "train['Same_day_travel']=train.Same_day_travel.astype('category')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CDuYCobcsFmW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting Airline to category as price varies with airlines ( Category column helps in tree based modelling)\n",
        "train.Airline=train.Airline.astype('category')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CWRW1UhHsFmb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating 2 columns weekday journey and weekday arrival that highligh what day of the week the flight took off and arrived\n",
        "# respectively. The idea is that the price will be marginally higher on weekends and this metric can help catch that.\n",
        "train['weekday_journey']=train['Date_of_Journey'].apply(lambda x: x.weekday())\n",
        "train['weekday_arrival']=train['Date_of_Arrival'].apply(lambda x: x.weekday())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Sl4GjiFvsFme",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating columns for year, month, and day of both departure and arrival as integers to be able to better \n",
        "# capture any seasonality due to date\n",
        "train['Year']=pd.DatetimeIndex(train['Date_of_Journey']).year\n",
        "train['month_of_journey']=pd.DatetimeIndex(train['Date_of_Journey']).month\n",
        "train['month_of_Arrival']=pd.DatetimeIndex(train['Date_of_Arrival']).month\n",
        "train['day_of_Arrival']=pd.DatetimeIndex(train['Date_of_Arrival']).day\n",
        "train['day_of_journey']=pd.DatetimeIndex(train['Date_of_Journey']).day"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EyCOkoqhsFmg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Seeing the number of columns generated - (11 more columns were created in the process)\n",
        "train.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UBuIbRErsFmi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Techniques that I tried but didnt help - Number of take offs and landings in a day \n",
        "# - Number of takeoffs by city and landings by city in a day\n",
        "\"\"\"r=train.groupby(['Date_of_Journey']).Date_of_Journey.apply(lambda x: x.count())\n",
        "train['Number_take-offs']=train.Date_of_Journey.map(r)\n",
        "r=train.groupby(['Date_of_Arrival']).Date_of_Arrival.apply(lambda x: x.count())\n",
        "train['Number_landings']=train.Date_of_Arrival.map(r)\n",
        "\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GvSxVqP1sFmk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"r=train.groupby(['Date_of_Journey','Source'])['Source'].count().reset_index(name='Number_takeoff_source')\n",
        "train=train.merge(r,how=\"left\", on=[\"Date_of_Journey\",\"Source\"])\n",
        "r=train.groupby(['Date_of_Arrival','Source'])['Source'].count().reset_index(name='Number_landing_Destination')\n",
        "train=train.merge(r,how=\"left\", on=[\"Date_of_Arrival\",\"Source\"])\n",
        "\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vmsG07rLsFmn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#train[(train.Date_of_Arrival==datetime.date(2019,3,1))][['Source','Number_landing_Destination']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3eyhYpCisFmr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"plt.subplots(figsize=(12,15))\n",
        "sns.boxplot(train[train.Source=='Delhi'].sort_values(['Date_of_Journey']).Number_takeoff_source,train[train.Source=='Delhi'].sort_values(['Date_of_Journey']).Price)\n",
        "plt.xticks(rotation=90)\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Gf0E25YXsFmv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Target Variable Normalization"
      ]
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "fFinykJvsFmw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sns.distplot(train[:len_train].Price, fit=stats.norm)\n",
        "l=plt.show()\n",
        "stats.probplot(train[:len_train].Price,plot=plt )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "bxg83AHFsFmy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "xt=np.log(train[:len_train].Price)\n",
        "sns.distplot(xt, fit=stats.norm)\n",
        "l=plt.show()\n",
        "stats.probplot(xt,plot=plt )\n",
        "\n",
        "train.Price=np.log(train.Price)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3QjRlNjWsFm1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Null Value Treatment"
      ]
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "nwIzwcE-sFm6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "##Identifying columns with missing values\n",
        "train.isnull().sum()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wvW4IbpmsFm9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Filtering data with missing value and filling up with appropriate values as determined\n",
        "train.loc[(train.Source=='Delhi') & (train.Destination=='Cochin') & (pd.DatetimeIndex(train['Date_of_Journey']).month==5)& (train.Duration==1420) &(train.Airline=='Air India') ].sort_values(['Date_of_Journey'])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YRjobYVusFm_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train.Route=train.Route.fillna('DEL → MAA → COK')\n",
        "train.loc[train.Total_Stops.isnull(),'Total_Stops']=1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "G15Id-hdsFnC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Outlier Treatment- Did not Use for modelling"
      ]
    },
    {
      "metadata": {
        "id": "S7msBwt5sFnD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"#Outlier Treatment\n",
        "x_out=pd.get_dummies(train[:len_train].drop('Price',axis=1))\n",
        "y_out=train[:len_train].Price\n",
        "\n",
        "enet= ElasticNet(max_iter=1000,alpha=0.001,l1_ratio=0)\n",
        "\n",
        "enet.fit(x_out,y_out)\n",
        "y_pred=enet.predict(x_out)\n",
        "resid=y_out-y_pred\n",
        "z=(resid-resid.mean())/resid.std()\n",
        "z=np.array(z)\n",
        "outlier=np.where(abs(z)>abs(z).std()*5)[0]\n",
        "\n",
        "plt.figure(figsize=(6, 6))\n",
        "plt.scatter(y_out, y_pred)\n",
        "plt.scatter(y_out.iloc[outlier], y_pred[outlier])\n",
        "plt.plot(range(7, 12), range(7, 12), color=\"red\")\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Yua2pLe8sFnF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"ridge=Ridge(alpha=1)\n",
        "\n",
        "ridge.fit(x_out,y_out)\n",
        "y_pred=ridge.predict(x_out)\n",
        "resid=y_out-y_pred\n",
        "z=(resid-resid.mean())/resid.std()\n",
        "z=np.array(z)\n",
        "outlier2=np.where(abs(z)>abs(z).std()*5)[0]\n",
        "len(outlier2)\n",
        "\n",
        "plt.figure(figsize=(6, 6))\n",
        "plt.scatter(y_out, y_pred)\n",
        "plt.scatter(y_out.iloc[outlier2], y_pred[outlier2])\n",
        "plt.plot(range(7, 12), range(7, 12), color=\"red\")\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4Efo0bwcsFnK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\"\"\"outlier_final=[]\n",
        "for i in outlier:\n",
        "    for j in outlier2:\n",
        "        if i==j:\n",
        "            outlier_final.append(i)\n",
        "(outlier_final)\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "cFo0eYrvsFnM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Fa_F7DiBsFnO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Skew Treatment (BOXCOX Transform)"
      ]
    },
    {
      "metadata": {
        "id": "uUzP7fuCsFnQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "numeric_dtypes = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n",
        "numerics2 = []\n",
        "for i in train.columns:\n",
        "    if i!='Price':\n",
        "        if train[i].dtype in numeric_dtypes: \n",
        "            numerics2.append(i)\n",
        "\n",
        "skew_features = train[numerics2].apply(lambda x: skew(x)).sort_values(ascending=False)\n",
        "skews = pd.DataFrame({'skew':skew_features})\n",
        "print(skews)\n",
        "\n",
        "high_skew = skew_features[abs(skew_features) > 0.3]\n",
        "high_skew = high_skew\n",
        "skew_index = high_skew.index\n",
        "\n",
        "for i in skew_index:\n",
        "     if i!='=Price':\n",
        "        train[i]= boxcox1p(train[i], boxcox_normmax(train[i]+1))\n",
        "\n",
        "        \n",
        "skew_features2 = train[numerics2].apply(lambda x: skew(x)).sort_values(ascending=False)\n",
        "skews2 = pd.DataFrame({'skew':skew_features2})\n",
        "skews2\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sfB9dzn7sFnS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data preperation for Modelling and Modelling"
      ]
    },
    {
      "metadata": {
        "id": "a_FJHuq4sFnU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating a sparse matrix using pandas get dummies function that helps split string columns into boolean values.\n",
        "df=pd.get_dummies(train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KYrxtttosFnW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Splitting test and train data\n",
        "train_df=df[:len_train]\n",
        "test_df=df[len_train:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZTiusO5RsFnY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Removing Price from independent variable data and keeping only price in the other\n",
        "x=train_df.drop('Price',axis=1)\n",
        "y=train_df['Price']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "89H2A6zbsFnd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#model training "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "A9dS69TvsFni",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Writing a function to evaluate model performance using KFOLD and cross validation\n",
        "def rmsle_cv(model):\n",
        "    kfold=KFold(n_splits=2, shuffle=True, random_state=10).get_n_splits(x)\n",
        "    rmse=np.sqrt(-cross_val_score(model,x,y,cv=kfold, scoring='neg_mean_squared_error'))\n",
        "    return rmse"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ziLwUPfSsFnj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# All paramaters used in the models have been tuned using the GRIDSEARCHCV funtion on the basis of articles on \n",
        "# Analytics Vidhya"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "et986ZLtsFnn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "lightgbm=LGBMRegressor(objective='regression',num_leaves=80,\n",
        "                              learning_rate=0.1, n_estimators=1200,\n",
        "                              max_bin = 30, bagging_fraction = 0.8,\n",
        "                              bagging_freq = 9, feature_fraction = 0.129,\n",
        "                              feature_fraction_seed=9, bagging_seed=9,\n",
        "                              min_data_in_leaf =3, min_sum_hessian_in_leaf = 6, random_state=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hN69eKh0sFnt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#rmsle_cv(lightgbm).mean(),rmsle_cv(lightgbm).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0aSRYI1ksFnw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "xgb=XGBRegressor(learning_rate =0.1, \n",
        "      n_estimators=1500, max_depth=12, min_child_weight=1,gamma=0, reg_alpha=2e-5,\n",
        "      subsample=0.8,colsample_bytree=0.8,\n",
        "      nthread=4,scale_pos_weight=1,seed=27,verbose=True,random_state=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KjRR0DkksFnz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#rmsle_cv(xgb).mean(),rmsle_cv(xgb).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9u2IbdkUsFn1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "grb=GradientBoostingRegressor(learning_rate=0.1,n_estimators=400, max_depth=12\n",
        "                              ,subsample=0.8,\n",
        "                              verbose=False,random_state=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ksvrFFbjsFn4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#rmsle_cv(grb).mean(), rmsle_cv(grb).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5YJ65PiPsFn7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "svr = Pipeline([('Scaler',RobustScaler()), ('SVR',SVR(C= 10000, epsilon= 0.008, gamma=0.009))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2EyxPXoSsFn9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#rmsle_cv(svr).mean(), rmsle_cv(svr).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "aiOlt735sFn_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "krr=KernelRidge(alpha=1, kernel='polynomial', gamma=0.001,degree=3,coef0=5)\n",
        "#rmsle_cv(krr).mean(), rmsle_cv(krr).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FmUjRTNlsFoA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "rf=RandomForestRegressor(n_estimators=3000, oob_score = False, n_jobs = -1,random_state =50,\n",
        "                         max_features = \"auto\", min_samples_leaf = 2,warm_start=True,criterion='mse',max_depth=50)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ra5mNZLOsFoB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#rmsle_cv(rf).mean(), rmsle_cv(rf).std() # Commented for faster running of complete code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "C72WFUz2sFoE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Final ensemble that uses outputs from each model regressor and the original data in the meta regressor XGB\n",
        "avg=StackingCVRegressor(regressors=(lightgbm,grb,svr,krr,rf),meta_regressor=xgb, use_features_in_secondary=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HPXZvxe0sFoH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Function to calculate RMSE error of the complete model for reference, similiar to rmsle_cv without KFOLD CV\n",
        "def rmsle(y, y_pred):\n",
        "    return np.sqrt(mean_squared_error(y, y_pred))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5xhFaRaesFoJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X=np.array(x)\n",
        "Y=np.array(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": false,
        "id": "clRwPqN3sFoL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Fitting the model on the train data and predicting the train price values\n",
        "avg.fit(X,Y)\n",
        "y_pred=avg.predict(X)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": true,
        "id": "xxHKaVVMsFoP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Plot of the model prediction with the original price values in the training data set to undestand model fit\n",
        "plt.scatter(list(y_pred),y)\n",
        "plt.plot(range(7,13),range(7,13),'r')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LXpFfA0msFoQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# RMSE value of the model on training data\n",
        "rmsle(y,y_pred)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "j4LmSWunsFoS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Predictions of test data\n",
        "Predict=avg.predict(np.array(test_df.drop('Price',axis=1)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nq5HpEBVsFoU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting price back to original scale and making it integer\n",
        "Predict=np.exp(Predict)\n",
        "Predict=Predict.astype(int)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tMxlW7qisFoX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Converting predicted values to dataframe\n",
        "Submission=pd.DataFrame(Predict,columns=['Price'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "q0pDR68HsFoY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "len(Submission)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "scrolled": false,
        "id": "4BxUPDI8sFoa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "Submission.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WVNYxWNCsFob",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Final Submission\n",
        "Submission.to_excel('Submission_rf.xlsx')"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
